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Abstract 


The Colorado Department of Education recently received state funding to implement a Student Re-Engagement 
Grant Program (SRGP). The goal of this project is to inform the administration of the SRGP by looking back at data 
from a previously administered grant program, called Colorado Graduation Pathways (CGP). The CGP program 
was a federally funded grant with similar objectives to the SRGP (e.g., decreasing the dropout rate; increasing the 
high school graduation rate). 


Student-level data from one year of this program were used in combination with state administrative data to 
describe the relationships between targeted interventions and educational outcomes for served students, such as 
staying in school and graduating from high school. This was not a causal study — that might suggest that 
outcomes were a result of specific interventions. Instead, the descriptive exploratory nature of the analyses is 
best used to generate ideas and guide conversations about strategic grant-making. The findings from the study 
suggest considering: 


e Engaging grantees in conversations about equity and access. 
e Expanding investments to create continuity through school transitions for all grade levels. 


e Targeting interventions and supports to students who change schools during 12" grade so that they are 
more likely to graduate. 


e Sustaining or increasing investments in Check & Connect to help keep students in school. 
e Accelerating investments in Title | and highly mobile students. 


e Requiring grantees to report program data at the student level. 
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Data Sources 


Administrative data from the Colorado Department of Education was utilized in this study. The data 
included student detail and demographic records for the 2014-15 to 2017-18 school years, as well as 
service utilization data collected for the Colorado Graduation Pathways (CGP) grant process for the 
2014-15 school year. 


Suggested Citation 


Clemens, E.V., & Shahan, L. (2019). The Colorado Dropout Prevention Framework: A Retrospective Analysis 
of Annual and Cohort Student Outcomes. (Report No. 106). Denver, CO: Colorado Evaluation and 
Action Lab at the University of Denver. 
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Introduction 


The Colorado Department of Education (CDE) recently received state funding to implement a Student 
Re-Engagement Grant Program (SRGP). The goal of the grant program is to assist local education providers 
in providing educational services and supports to maintain student engagement and facilitate student 
re-engagement at the secondary level. In order to tailor SRGP and maximize the likelihood of positive 
student outcomes, a retrospective analysis of data from the Colorado Graduation Pathways (CGP) was 
used to identify relationships between aspects of the Colorado Dropout Prevention Framework (CDPF) and 
student outcomes. 


Colorado Graduation Pathways 
Prior to the launch of SRGP, a federal grant program called Colorado Graduation Pathways (CGP) was 


implemented by CDE, with a goal of developing dropout prevention models that were both sustainable 
and replicable. With an additional focus on improvement of interim indicators related to graduation 
(e.g. attendance, behavior, and course completion). CGP-funded schools selected methods and tactics 
from CDPF; some were used universally in the school while others were targeted to serve specific 
students. 


Colorado Dropout Prevention Framework 
CDPF is a framework intended to ensure that all students have educational opportunities and effective 


academic guidance to attain their educational goals. This is accomplished by focusing on three essential 
elements: 1) Identification, 2) Intervention & Support, and 3) Institutional Change, supported by the 
following methods and tactics: 


e Data Analysis e Family Partnering 

e Early Warning Systems e Transition Programs 

e Tracking Out-of-School Youth e Multiple Pathways to Graduation 

e Assess and Enhance School Climate e Re-engagement of Out-of-School Youth 
e Policy and Practices Review e Enhanced Counseling and Mentoring 

e Community Engagement e Credit Recovery Options 


This study is a first step toward understanding the relationship between the use of CDPF methods and 
tactics and students’ progress in school. The focus of this study is primarily on the targeted interventions 
that were delivered to students as part of a CDPF method and tactic. In the current study, information was 
available regarding student participation in five interventions (see Figure 1). Findings from the proposed 
analyses are intended to be validated in later years using current and future SRGP data, and ultimately 
inform development of the next SRGP request for proposals. 


Figure 1 
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Description of the Study 


The study is descriptive and exploratory in nature. CGP program data were first used in combination with 
school-wide demographics to determine if some groups of students received CGP interventions at 
disproportional rates. Then, the CGP intervention data were compared to data on how students exited the 
academic year. School outcomes were categorized by CDE to group end-of-year outcomes into positive, 
neutral, or negative outcomes. End-of-year positive outcomes include graduation, obtaining a GED, or 
continuing with school. Neutral outcomes include transfers, illness/injury, or death. Negative outcomes 
include aging out of services, transferring to a detention facility, expulsion, extended absence, or dropping 
out. Together, this information can be used to inform future investments in dropout prevention and 
student re-engagement services in Colorado. 
) PART ONE: Describes the context for understanding the relationship between 
interventions and outcomes by focusing on CGP dosage services, disproportionality 
in targeted intervention participation, and school mobility. 


PART TWO: Describes annual — or end-of-year outcomes — for students who 
received a targeted CGP intervention in Grades 9, 10, 11, and 12 during the 
2014-15 school year. 


PART THREE: Describes outcomes for the Class of 2018 cohort. The analyses pertain 
to students who received a CGP-targeted intervention as ninth graders during the 
| 2014-15 school year. 


Methods 


Sample 


There were two samples for used in this study. The annual sample includes students in grades nine 
through twelve who were served by CGP in the 2014-15 school year. The cohort sample is the students in 
the Class of 2018 who were served by CGP during the 2014-15 school year as ninth graders. Data from the 
2014-15 school year were used because that is the year in which CGP-funded schools were required to 
report to CDE the specific interventions delivered to individual students. 


Annual Analysis 


The sampling frame was defined as students who were reported by CGP interventions as having received 
services under the grant in 2014-15 and whose reported State Assigned Student Identifier (SASID) also 
matched records in CDE’s end-of-year data collection (Figure 2). A total of 6,366 unique students were 
included in the served students sample and 36,586 unique students who attended a CGP school were 
retained for proportional comparison. 
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Figure 2 
Gatceory Type CGP Served Percentage of 
Students Sample 

Student Grade 9" Grade 1,180 19% 
10" Grade 1,097 17% 
11" Grade 1,674 26% 
12" Grade 2445 38% 

Gender Female 2,886 45% 
Male 3,480 55% 

Race/Ethnicity African American 806 13% 
Asian American 180 3% 
Caucasian 1,680 26% 
Hawaiian/Pacific Islander 18 0% 
Hispanic 3,409 54% 
Native American 118 2% 
Two or More 155 2% 

Unique Student ESL/ELL 2,156 34% 

Populations Free or Reduced Lunch (FRL) 3,888 61% 
Highly Mobile 530 8% 
SPED Status 556 9% 
Title | 714 11% 

Total Number of Students 6,366 100% 


A subset of the annual analysis sample was used for the cohort analysis. The sample was selected using 
students who were reported to be in the ninth grade in the 2014-15 school year and did not have a ninth 
grade or higher record the previous year (2013-14 school year). A total of 1,079 unique students were 
included in the served student cohort sample (Figure 3). The same method was used to create a 
comparison group of students who attended a CGP school with a total of 9,907 unique students. 


Figure 3 
eatceny Tpe CGP Served Percentage of 
Students Sample 
Gender Female 497 46% 
Male 582 54% 
Race/Ethnicity African American 170 16% 
Asian American 17 2% 
Caucasian 256 24% 
Hispanic 582 54% 
Native American 23 2% 
Two or More 30 3% 
Unique Student ESL/ELL 372 34% 
Populations Free or Reduced Lunch (FRL) 754 70% 
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Note. Due to relatively small sample size, the Hawaiian/Pacific Islander category will not be reported in 


individual student may fit into multiple categories. 


Data Cleaning 


Analysis of administrative data sets often involves making some decisions while cleaning the data. In this 
study, data from student demographic files were collapsed so that only one record per student was 
retained. Where there were multiple and/or conflicting records, the records were combined so that any 
field with a positive indicator was retained over a negative (e.g. homelessness, various race categories, 
etc.), such that a student with a record indicating “Yes” for Asian American and another record indicating 
“Yes” for Caucasian would have a final record with “Yes” for both fields. CDE federal race reporting 
standards were then reapplied to the consolidated data to update that field; 2,815 records were 
combined. 


Unique student populations were calculated as follows: 


e English as a Second Language/ English Language Learner (ESL/ELL): A “Yes” indicator in the ELL 
field or a value greater than 1 in the ESL field 


e Free or Reduced Lunch (FRL): A value of “F” or “R” in the FRL field 


e Highly Mobile: A positive indicator in the Homeless or Refugee fields, or a value of “B”, “C”, or “D” 
in the Migrant field, or the student was on a list of foster care students 


e Special Education (SPED) Status: a positive indicator in the SPED field 


e Title |: a positive indicator in the Title | field. 


A similar consolidation process was used for students who had multiple CGP service records with 210 
records combined. When comparing student demographic data to the CGP grant collected data, only 
records where demographic and CGP data could be linked were kept as the CGP-served population — 465 
records were excluded. Of the 1,079 students in the ninth-grade cohort, 80% had a 2017-18 school detail 
record, all were retained, and the last detail record for the student (regardless of year) was used for the 
final outcome analysis. 


Analytic method 


The majority of the analyses were descriptive statistics. Comparisons were assessed via chi-square tests of 
association. All comparisons highlighted in the narrative were significant at p < .05 level, unless noted. All 
statistical comparisons are available upon request. 
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Recommendations for Student Re-Engagement Grant 
Program Grant-Making 


Recommendations for SRGP grant-making are presented in this section along with the key supporting 
findings. The detailed results associated with each research question are presented later in this report. 


Engage grantees in conversations about equity and access. 


Male students were overrepresented in CGP interventions. 


African American students were overrepresented in CGP interventions, and 
Hispanic students were underrepresented. 


ESL/ELL students were underrepresented in CGP interventions, and Title | 
students were overrepresented. 


Analysis of the CGP program data suggest that some student groups received disproportional access to 
services, meaning that the proportion of students in a group receiving the targeted intervention was 
different than the proportion of that student group in the CGP school population. Disproportional 
representation in CGP intervention services may have been intentional and data-driven. There could also 
be systematic gaps in how students were identified for targeted interventions. The current study is only 
designed to describe the disproportionality and pose recommendations for where to engage in meaningful 
conversations with grantees. Engaging former and future grantees in conversations about their 
identification process through an equity lens may elucidate opportunities to ensure equitable access to 
services. 


Expand investments to create continuity through school transitions for all grade 
levels. 


Eighty-nine percent of students who enrolled in a CGP intervention service at the 
start of the academic year and did not transfer out had a positive outcome at the 
end of the school year. 


Whereas, 73% of students who moved schools had a positive outcome that same 
school year. 


Following students for four years after initially entering ninth grade highlights the 
relationship between school moves and fewer positive outcomes. Students who 
transferred out of a CGP school graduated at lower rates than students who 
transferred into a CGP school (50% and 67%, respectively). 
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Regardless of intervention type or grade level, students who 
remained enrolled in the same CGP school throughout the 

2014-15 school year had more positive outcomes than those 
who changed schools. Remaining in school 


Positive outcomes, as defined by CDE, 
refer to at the end of the year: 


Obtaining a high school 
credential (e.g., GED) 


A closer look at the data on school moves suggests CGP 

schools may have developed successful onboarding practices 
that could be replicated and expanded. More attention is Graduating from high school 
needed on strategies to support students as they exit, 
particularly to schools who are not receiving grant funds to Or at any point during the year: 


implement CDPF. : 
men e Returning to school after a 


prolonged absence 


Students who transferred into a CGP school had 81% positive 
outcomes, students who transferred between schools had 
71% positive outcomes, more than two transfers had 65% positive outcomes, and students who 
transferred out of a CGP school had 55% positive outcomes that same school year. 


Target interventions and supports to students who change schools during 12‘ 
grade so that they are more likely to graduate. 


Fifty-five percent of all 12'° graders who received CGP services graduated. If the 
student remained at the same CGP school, that number rose to 60% and dropped 
_ to 35% for 12" grade students who transferred schools. 


When 12" graders change schools, even into another CGP school, they are less likely to graduate that year 
than their CGP-served peers who remained in the same high school. Students who did change schools 
were more likely to graduate that year if they participated in attendance interventions or credit recovery 
programming. 


Given low graduation rates for highly mobile students statewide and within CGP schools, SRGP funds 


might also be wisely invested in policy and practice reviews that build the capacity of schools to implement 
the educational stability protections for students who experience homelessness or foster care. 


Sustain or increase investments in Check & Connect to help keep students in 
school. 


Nearly 100% of students who received Check & Connect services remained in 
school, compared to 60% for all other students who received CGP services. 


Only a small number of students received this service and they were 
disproportionally male and Caucasian. 
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Check & Connect is an evidence-based intervention’ and these exploratory analyses suggest that delivery 
of this intervention by CGP schools is associated with staying in school. Although only 228 students 
between ninth and 11" grade received Check & Connect services, 227 of those students had a positive 
outcome at the end of the school year. An area for further exploration in grant planning and making is 
learning more about the selection of students for this program. Under the CGP grant, the students served 
with Check & Connect were disproportionally male and Caucasian’. Conversations with Colorado schools 
that use Check & Connect might inform the populations for whom this intervention may work best or 
could be intentionally expanded to serve. 


Accelerate investments in Title | and highly mobile students 


i Only 46% of Title | students remained in school. 
Only 20% of Title | 12" graders who received services graduated in 2014-15. 
Only 44% of highly mobile 12" graders who received services graduated in 2014-15. 


Forty-six percent of Title | students had lower rates of staying in school than their peers, compared to 63% 
of all CGP students. Title | students and highly mobile students also had a much lower graduation rate than 
their peers receiving CGP services. 


Title | students received attendance and dropout recovery interventions at a higher rate than others, 
accounting for 49% of all dropout recovery services. This group did not receive Check & Connect services 
and had disproportionally low rates of credit recovery and graduation coaching services. Highly mobile 
students also had higher rates of receiving attendance and dropout recovery interventions and lower rates 
of Check & Connect. 


Require grantees to report program data at the student level 


A primary limitation of this study was knowing only what intervention students 
participated in during one school year. Many of these students continued in CGP schools 
for multiple school years, thus the CGP dosage information is incomplete and findings 
_ | can only be used with caution. 
When the CGP program was implemented, CDE was in the process of building the infrastructure for 
grantees to report program information by SASID. That capacity has been built, thus requiring student 
level reporting is now more feasible. 


1 Sinclair, M. F., Christenson, S. L., & Thurlow, M. L. (2005). Promoting school completion of urban secondary youth 
with emotional or behavioral disabilities. Exceptional Children, 71(4), 465-482. 

* Although Figures 12 and 15 illustrate gender and race separately, the disproportionality also applies to the 
combination of male and Caucasian. 
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PART ONE: CONTEXT — DOSAGE, DISPROPORTIONALITY, 
AND SCHOOL MOBILITY 


Research Question 1: What and how many methods, tactics, and targeted 
interventions did students in CGP schools receive? What were the most typical 
combinations? 


Research Question 2: What disproportionality exists in service delivery? 


Research Question 3: On an annual basis, what were the school mobility patterns 
for students served by CGP interventions? 


Research Question 4: For the cohort of students served by CGP interventions in 
ninth grade, what were the school mobility patterns throughout their high school 
experience? 


Each research question was examined for all students receiving CGP services and by unique student 
groups (i.e., demographic characteristics and Instructional Service Program Types). In part two of this 
report, some of the above questions will be expanded to a cohort approach and incorporate four years of 
data since initially entering a CGP school in ninth grade. 


The student level data reported were a list of students served and the interventions received. No details 
about level of participation in a given intervention or length of engagement in the intervention were 
reported. These data reported by CGP schools were correlated with records from CDE’s End-of-Year data 
collection to answer the four research questions below. 


Results 


Research Question 1: What and how many methods, tactics, and targeted 
interventions did students in CGP schools receive? What were the most typical 
combinations? 


CGP dosage is a combination of the methods and tactics that were a focus of the schools in a given year 
and the targeted interventions a given student received. Many of the methods and tactics can be 
considered ‘universal services’ because they are used to identify students and facilitate institutional 
change. Others are categories of interventions and supports that house the menu of targeted 
interventions that individual students received. 


Dosage: methods and tactics (school level) 


CGP schools reported implementing on average 6.8 methods and tactics concurrently. Nearly two-thirds of 
students in the study, 63%, attended a school where six or more methods and tactics were a focus during 
the 2014-15 academic year. The most commonly reported methods and tactics were Credit Recovery 
Options, Data Analysis, Tracking Out-of-School, Assess and Enhance School Climate, and Multiple 
Pathways to Graduation (see Figure 4). Transition interventions were by far the least frequently used 
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method and tactic. Data were not provided by the CGP schools to CDE indicating if a given method or 
tactic was truly universally implemented or touched subsets of the school population. 


Figure 4 


Number of Schools Providing Each Universal Service 


jo) 


5 10 15 20 30 


i) 
nn 


* Credit Recovery Options * 

Data analysis 

Assess and Enhance School Climate 

* Multiple Pathways to Graduation * 

* Enhanced Counseling and Mentoring * 
Tracking Out-of-School Youth 

Policy and Practices Review 

Community Engagement 

* Early Warning Systems * 
Re-engagement of Out-of-School Youth 
Family Partnering 


Transition Interventions 


Note. The methods and tactics that were reported at the student level and have targeted interventions 
associated with them are denoted with an asterisk. 


Dosage: targeted interventions (annual sample) 


The average number of services received by students was 1.49, with 63% of students receiving at least one 
intervention. More than half of CGP students received either attendance or credit recovery interventions 
(Figure 5). Check & Connect was the least frequently implemented intervention, with only 4% of students 
receiving this intervention (Figure 6). 


Targeted interventions were not evenly dispersed across grade levels. As noted in the description of the 
sample (see p. 8), 38% of the students who received one or more interventions were in 12" grade, 26% 
were 11" graders, 17% tenth graders, and 19% ninth graders. Due to 12" grade students’ 
overrepresentation in the overall programming, they account for the majority of slots in most targeted 
interventions and received three or more interventions during the same academic year at a higher rate 
than all other grades combined (Figure 5). 


A closer look at the interventions delivered by grade level might inform future guidance on balancing 


interventions across grade levels (Figures 7 and 8). Credit recovery was the most frequently delivered 
targeted intervention to first-time ninth graders. 
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Figure 5 i 
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Figure 7 
Intervention Dosage by Grade 
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Dosage: targeted interventions (cohort sample) 


For the ninth-grade cohort, the average number of interventions was 1.16, with 85% of students receiving 
one intervention and 15% receiving more than one. 


Figure 8 


Interventions Received Cohort Sample, During 9th Grade 
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Note. Information is not available on the interventions students in the cohort sample received after the 
school year when they initially entered ninth grade. 
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Combinations of methods, tactics, and targeted interventions 


A crosswalk of methods and tactics reported at the school level with targeted interventions reported at 
the student level for the annual sample, which includes all high school grade levels (Figure 9), and the 
cohort sample, which includes only first-time ninth graders (Figure 10), illustrates patterns in service 
combinations. 


The attendance interventions were most common in schools where multiple pathways to graduation and 
credit recovery services were available. All schools offering Check & Connect indicated that they also used 
data analysis, school climate, and credit recovery services. The credit recovery interventions were most 
common at schools where school climate services, community involvement, pathways, and credit recovery 
services were available. Dropout recovery interventions were most common at schools that indicated 
re-engagement and credit recovery. Graduation coaching interventions were typically found in 
combination with multiple methods. 


Check & Connect was conceptualized as part of the Enhanced Counseling and Mentoring method and 
tactic (see Figure 1 on p 7). Figures 9 and 10, however, suggest that schools might conceptualize this 


targeted intervention differently, perhaps as Data Analysis. 


Figure 9 


Attendance | Check & Credit Dropout | Graduation 


Annual Analysis Intervention Connect Recove Recove Coaching 


Data analysis 

Early Warning Systems 

Tracking Out-of-School Youth 
Assess and Enhance School Climate 


Policy and Practices Review 
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Transition Interventions 


Multiple Pathways to Graduation 


Re-engagement of Out-of-School Youth 


Enhanced Counseling and Mentoring 


Credit Recovery Options 
Figure 10 
Attendance i Dropout | Graduation 

9 Grade Cohort Intervention 

Data analysis 23% 

Early Warning Systems 23% 

Tracking Out-of-School Youth 20% 

Assess and Enhance School Climate 29% 

Policy and Practices Review 19% 

Community Engagement 14% 
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Attendance | Check & Credit Dropout | Graduation 
9 Grade Cohort Intervention | Connect | Recovery Recovery Coaching 
Family Partnering 23% 0% 29% 5% 
Transition Interventions 19% 0% 
Multiple Pathways to Graduation es ox 


Re-engagement of Out-of-School Youth 14% 0% 
Enhanced Counseling and Mentoring 29% 0% 


Credit Recovery Options 
Note. Two schools did not report on methods and tactics, which may explain some findings, such as 86% 
of the students who received credit recovery services attended a school that offered Credit Recovery 
Options services. 


Research Question 2: What disproportionality exists in service delivery? 


Examining disproportionality through a descriptive lens can guide conversations about equity and access. 
In particular, this information might be used in combination with data on dropout rates and graduation 
rates. 


When disproportionally high participation in target services mirrors the need to 
accelerate progress based on historical data, that is a good indicator of programs 
being targeted at known need. 


Conversely, lack of disproportional participation in services when there is a known 
need to accelerate progress might prompt exploring opportunities to expand 
investments in the future. 


Gender 


Female students were underrepresented and male students were overrepresented in the general service 
population. This disproportionality in service delivery aligns with the statewide data that indicate that 
males consistently dropout at a higher rate than females, which suggests a need to deliver targeted 
interventions to proportionally more males?. 


3 Colorado Department of Education (2016). 2014-15 State Policy Report: Dropout Prevention and Student 
Engagement. Retrieved from the Colorado Department of Education website: https://www.cde.state.co.us/dropout 


prevention/2015dropoutpreventionpolicyreport 
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Figure 11 


Gender Percentages 
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The emphasis on providing targeted male students was not evenly distributed across interventions; Check 
& Connect and dropout recovery interventions were provided most often to males. Female students only 
accounted for 38% and 41% of Check & Connect and dropout recovery interventions, respectively. 


Figure 12 
Gender Across Interventions 
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Note. Graduation coaching was the only intervention where the differences in participation by gender were 
not statistically significant. 


Although most of the gender differences in the ninth-grade cohort were not statistically significant, Figure 
13 is included because patterns diverged from the annual sample. Credit recovery was the only 
intervention where the higher proportion of males was statistically significant. The lack of statistical 
significance here is likely due to small sample sizes within some of the interventions (e.g. dropout recovery 
with only 21 students). 
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Figure 13 7 
Cohort - Gender Across Interventions 
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Race/Ethnicity 


African American students and Native American Students* were overrepresented, while Hispanic students 
were underrepresented in overall service delivery. The statewide data on dropout rates determine that 
Hispanic students are also a group who would benefit from more targeted interventions as their dropout 
rates are consistently above the state average’. 


Figure 14 
Race/Ethnicity Percentages 
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Note. *The Native American findings of overrepresentation was not statistically significant, but that is 
likely a reflection of sample size. Practically the emphasis on providing services to Native American 
students is clear in the data. 


4 Colorado Department of Education (2016). 2014-15 State Policy Report: Dropout Prevention and Student 
Engagement. Retrieved from the Colorado Department of Education website: https://www.cde.state.co.us/dropout 


prevention/2015dropoutpreventionpolicyreport 
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A closer look at how the disproportionality in service delivery is 
dispersed across interventions can guide conversations that may 
inform future program planning and investments. African 
American students were overrepresented in attendance 
interventions and underrepresented in all other interventions. 
Caucasian students were overrepresented in Check & Connect and 
graduation coaching. The overrepresentation of Caucasian 
students in Check & Connect is particularly noteworthy because it 
is the approach with the strongest evidence base and this group of 
students tends to have low dropout rates. Hispanic students were 
underrepresented in attendance interventions, Check & Connect, and graduation coaching, with a slight 
overrepresentation in both credit and dropout recovery services. Native Americans accounted for 27% of 
the total students receiving Check & Connect (61 of 228 total) and were slightly underrepresented in all 
other interventions except credit recovery. 


Throughout the report, a large 
gray bar is used as an anchor for 
chart comparisons. 


This bar represents the overall 
group population, which gives 
context for various breakout 
comparisons. 


Figure 15 
Race and Ethnicity Across Interventions 
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lm Credit Recovery 10% 2% 28% 56% 2% 2% 
m Dropout Recovery 8% A% 30% 55% 1% 1% 
m Graduation Coaches 7% 1% 40% A7% 1% 3% 


Race and ethnicity showed differing patterns for the ninth-grade cohort compared to the overall annual 
sample. Race and ethnicity differences across the interventions were significant for attendance 
interventions, Check & Connect, and credit recovery interventions. 
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Figure 16 
Cohort - Race/Ethncity Across Interventions 
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Unique Student Populations 


Across the various unique student populations, ESL/ELL students and special education students were 
underrepresented. Title | students were overrepresented. The overrepresentation of Title | students aligns 
with historical need to improve the dropout rates for these unique population of students (Colorado 
Department of Education, 2016). More emphasis on providing services to highly mobile students is needed 
because their dropout rates tend to the highest statewide (Colorado Department of Education, 2016). 


Figure 17 
Unique Population Percentages 
a 61% 61% 

60% 
50% 42% 
40% 34% 
30% 
20% 
10% 
0% 


8% 8% 9%. 1% We 9% 


ESL/ELL FRL Highly Mobile SPED Title | 


m Served Students m All CPG School Students 


www.ColoradoLab.org 22 


Colorado Evaluation and Action Lab 


The disproportionality in delivery of targeted interventions suggests a tendency to focus on reactive rather 
than preventative services for ESL/ ELL, highly mobile, and Title | student populations. ESL/ELL students 
were overrepresented in dropout recovery and were underrepresented in all other interventions except 
for credit recovery. Students who qualified for an FRL were underrepresented in credit recovery and 
graduation coaching. Highly mobile students were underrepresented in Check & Connect (approaching 
significance; chi-square 2.905, p=.088) and overrepresented in dropout recovery services. Title | students 
did not receive Check & Connect services and were underrepresented in credit recovery and graduation 
coaching. This group was overrepresented in both the attendance interventions and dropout recovery, 
accounting for almost 50% of students in dropout recovery. 


Whereas, for students eligible for special education services, data suggests a more concerted focus on 
dropout prevention. Special education students were overrepresented in Check & Connect (approaching 
significance; chi-square 2.866, p=.09) and underrepresented in dropout recovery services. 


Figure 18 
Unique Student Popluations Across Interventions 
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For the ninth-grade cohort, the distribution of unique student populations across and within the 
interventions was similar to the annual sample. 
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Figure 19 : 


Cohort Unique Student Populations Across Interventions 
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Research Question 3: On an annual basis, what were the school mobility patterns 
for students served by the CGP interventions? 


The majority of CGP-served students remained in the same 
school throughout the 2014-15 school year, with only 4% 
moving two times or more. 


Of the students who changed schools once, 34% moved 
between CGP schools, 15% transferred into a CGP from a 
non-CGP school, and 53% transferred out of CGP school to 
a non-CGP school. 
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Figure 20 
Number of School Changes 
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Ninth and 10" grade students. Ninth grade students had the highest rate of any transfer while 10" 
graders had the lowest proportion of students who transferred out of a CGP school and into a non-CGP 
school. Twelfth graders had the largest proportion of students who remained in one CGP school. 


Figure 21 
Student Grade by Number of School Changes 
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In school movements, compared to sample percentages, a higher proportion of female students 
transferred between CGP schools or attended more than two schools. The differences were not 
statistically significant when looking at the number of school changes. 


www.ColoradoLab.org 25 


Colorado Evaluation and Action Lab 


Figure 22 : 
Gender by School Movements 
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Figure 23 
Gender by School Changes 
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Race/Ethnicity 


During the 2014-15 school year, African American students were 
most likley to remain in the same CGP school. Caucasian students 
were most likely to transfer into a CGP school. Hispanic students 
had the highest rates of transfer between CGP schools as well as 
out of a CGP school and multiple school changes. 


Figure 24 
Race/Ethnciity Across School Movment Patterns 
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Figure 25 
Race/Ethnicity Across School Changes 
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Unique Student Populations 


Title | and highly mobile students had the highest rate of multiple school movements. ESL/ELL and Title | 
students who experienced one school change had disproportionally high rates of transfers between CGP 
schools. 


Figure 26 
Unique Student Populations Across School Movement Patterns 
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Note. Some students may be represented in multiple unique populations (e.g., a student could be both 
highly mobile and eligible for special education services). 
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Figure 27 
Unique Student Populations Across School Changes 
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Research Question 4: For the cohort of students served by CGP interventions in 
ninth grade, what were the school mobility patterns throughout their high school 
experience? 


The majority of CGP-served students remained in the same School movement was assessed as 


school throughout the 2014-15 to 2017-18 school years enrollment during the 2014-15 to 2017- 
(44%) or had one school move (26%). Compared to the 18 academic years: 

annual sample, a larger proportion attended more than two © OneCGP school 

schools (31%). Fifty-eight percent remained in a CGP school. 

Less than 1% of students did not show a CGP school in the e Moved from a non-CGP school 
school detail records and are not displayed. The range of to a CGP school 

school changes was 0-14, with an average of 1.25 changes. Moved from a CGP school to a 


non-CGP school 


More than two schools attended 
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Figure 28 : 
Cohort School Movement Patterns 
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Figure 29 
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Gender 


Gender differences were not statistically significant for either school movement patterns or school 
changes. 


Figure 30 
Cohort - Gender by School Movements 
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Figure 31 


Cohort - Gender by School Changes 
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Race/Ethnicity 


During the 2014-15 to 2017-18 school years, African 
American students were overrepresented in moving 
between CGP schools. Caucasians were 
overrepresented in remaining in the same CGP 
school and in moving into a CGP school; they wer 
underrepresented in moving between CGP schools. 
Hispanic students were overrepresented in moving between CGP schools, moving out of CGP schools, and 
had the highest rate of multiple transfers. 


Figure 32 
Cohort - Race/Ethnicity Across School Movement Patterns 
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Figure 33 
Cohort - Race/Ethnicity Across School Changes 
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Unique Student Populations 


ESL/ELL students had the highest rate of transfer between CGP schools. ESL/ELL and FRL students had the 
lowest rate of transfer into a CGP school as well as out of a CGP school. Title | students had the highest 


rate of school transfers. 


Figure 34 
Cohort - Unique Student Populations Across School Movement Patterns 
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Note. Some students may be represented in multiple unique populations (e.g., a student could be both 
highly mobile and eligible for special education services). 
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Figure 35 
Cohort - Unique Student Populations Across School Movement 
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PART TWO: ANNUAL OUTCOMES 


Research Question 5: What are the patterns among targeted interventions, school 
mobility, and student outcomes at the end of the school year? 


Although students may have been exposed to or served behind 
the scenes by a variety of methods and tactics, only the targeted 
interventions were reported at the student level. Thus, in this 
section, only the targeted interventions are separated from the 
methods and tactics analyses. 


Throughout this report, 
differences stated in the narrative 
(e.g., “higher rate”; 
“overrepresented”) were 
statistically significant at <.05 
level. Whereas language implying 
rank order (e.g., “highest 
percentage”) is descriptive and 
does not apply a statistically 
significant difference. 


Comparisons are also drawn in this section between school 
mobility and student outcomes because targeted interventions 
were implemented by CGP schools. Connecting school mobility 
patterns to student provides important context and can inform 
future investments in dropout prevention and student 
re-engagement. 


The specific test-statistics for each 
comparison are available upon 
request. 


School outcomes were categorized by CDE to group end-of-year 
outcomes into positive, neutral, or negative outcomes. End-of- 
year positive outcomes include graduation, obtaining a GED, or 
continuing with school. Additionally, if at any time during the school year a student returned from school 
after an extended absence, this is coded as a positive outcome. Neutral educational outcomes include 
transfers, illness/injury, or death. Negative outcomes include aging out of services, transferring to a 
detention facility, expulsion, extended absence, or dropping out. 


The outcomes across all of the interventions was overwhelmingly 
positive with 85% of students ending the 2014-15 school year with a 
positive outcome (Figure 36). Twelfth graders accounted for 59% of 
the negative outcomes, which is much higher than their percentage in 
the population served (38%) [see Figure 37]. 


Throughout the report, a large 
gray bar is used as an anchor 
for chart comparisons. 


This bar represents the overall 
group population, which gives 
context for various breakout 
comparisons. 
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Figure 36 : 
Outcomes for CPG Served Students 
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Outcomes Across Grade Level 
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Outcomes by Student Characteristics 


Outcomes across gender were not significant. African American students had a higher rate of positive 
outcomes, while Hispanic and Caucasian students had higher rates of both neutral and negative outcomes. 
Title |, ESL/ELL (approaching significance; chi-square 4.916, p=.086), and highly mobile students were 
overrepresented in both negative and neutral outcomes. 
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Figure 38 : 
Outcomes Across Race/Ethnicity 
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Outcomes Across Unique Student Populations 
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Comparison of Outcomes to School Mobility 


Students who were able to remain in any CGP school had the highest percentage of positive outcomes. 
Students who transferred out of a CGP to a non-CGP school had the highest number of neutral outcomes. 
Students who remained in the same school had the highest rate of positive outcomes, while the rate of 
neutral and negative outcomes increased with transferring schools. 
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Figure 40 * 
Outcomes Across School Movement Patterns 
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Notes. Statistical comparisons were not made among these categories. 


Figure 41 


Outcomes Across School Changes 
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Outcomes and Services Received 


Applying descriptive findings to programmatic decisions must be done with extreme caution. These 
analyses do not take into account the unique circumstances of the students receiving those services. 
Higher rates of positive outcomes are not necessarily a function of intervention effectiveness. 


Students who received Check & Connect had the highest number of positive outcomes (227 of 228 — 100% 
due to rounding in Figure 42). Credit recovery and attendance interventions had the highest percentage of 
positive outcomes across a large number of students served. Students receiving dropout services or 
graduation coaching had the highest rate of negative outcomes. 


Figure 42 
Outcomes Across Interventions 
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Students who attended schools with community involvement, transitions, and credit recovery had the 
highest rates of positive outcomes. Additionally, the lowest rates for negative outcomes were at schools 
providing transition and credit recovery services. These methods and tactics findings should be used with 
caution because these were implemented at the school a given student attended, but it is unclear if that 
student was engaged in the methods or tactics. 
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Figure 43 5 
Outcomes Across Methods and Tactics 
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Comparison of Types of Positive Outcomes 


Positive outcomes were broken out into returned to school after 
an extended absence (at any point in the year), stayed in school, 
and graduation for 12" graders. 


A total of 393 students returned after six weeks, with the majority 
being 12" graders. The most common positive outcome was 
remaining in school, with 4,027 students. Finally, 1,321 twelfth 
grade students who received services graduated in the 2014-15 
school year for a 55% graduation rate. Of note, students who 
returned to school after an extended absence were also counted 
in final outcomes (e.g. a student can be counted in the returned 
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to school category and also graduate). End-of-year positive outcomes include graduation, obtaining a GED, 
or continuing with school. 


Additionally, for this report, if at any time the student has returned from school after an extended 
absence, this is coded as a positive outcome. Neutral outcomes include transfers, illness/injury, or death. 
Negative outcomes include aging out of services, transferring to a detention facility, expulsion, extended 
absence, or dropping out. 


THE SPECIFIC POSITIVE OUTCOMES ARE ONLY REPORTED WHEN THERE WAS SUFFICIENT SAMPLE SIZE 
TO DO SO. FOR EXAMPLE, THERE WERE NOT ENOUGH STUDENTS WHO EARNED A GED TO REPORT 


INFORMATION ON THAT OUTCOME. 


Figure 44 
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Note. Students could have a positive outcome of returned to school and stayed in school OR graduated. 
The denominator for returned to school and stayed in school is the full students served 2014-15 annual 
sample, whereas the denominator for 12"" graders who graduated is students who were enrolled in 12" 
grade during the 2014-15 school year. 


Examining the positive outcomes by intervention received, 227 of 228 students who received the Check & 
Connect intervention remained in school at the end of the school year. The attendance and credit 
recovery interventions had high rates of students remaining in school for a larger population, both with 
over 2,000 students remaining in school who received that service. Those two interventions had the 
highest rates of 12" grade students who graduated, though the attendance intervention was not 
statistically significant. 
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Figure 45 


Positive Outcomes by Intervention 
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Students who attended schools that provided tracking, family partnering, and enhanced counseling had 
the highest rates of returning to school. Students who attended schools that indicated they used the 
following methods and tactics had the highest rates of staying in school; data analysis, early warning, 
transitions, pathways, and credit recovery services. Twelfth grade graduation rates were highest for 
students who attended schools providing community engagement, transitions, pathways, and credit 
recovery. 
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Figure 46 
Positive Outcomes by Methods and Tactics 
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Comparison of Different Positive Outcomes by Student Characteristics 


There were no noticeable differences in the various positive outcomes by gender. Native American 
students had the highest rate of remaining in school (84%), followed by African American students at 71%. 
African American students had a higher graduation rate than their peers (71%). 
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Figure 47 
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Figure 48 


African American 

Asian American 
Caucasian 
Hawaiian/Pacific Islander 
Hispanic 

Native American 


Two or More 


Highly mobile and Title | students had higher rates of 
returning to school. FRL and special education students 
had the highest rates of staying in school. Title | students 
had the lowest rates of staying in school and the lowest 
graduation rate for 12" graders. 


www.ColoradoLab.org 


Staying In School by Race/Ethnciity 
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Throughout the report, a large gray bar is 
used as an anchor for chart comparisons. 


This bar represents the overall group 
population, which gives context for various 
breakout comparisons. 
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Figure 49 
Positive Outcomes Across Unique Student Populations 
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Note. Students could have a positive outcome of returned to school and stayed in school OR graduated. 
The denominator for returned to school and stayed in school is the full students served 2014-15 annual 
sample, whereas the denominator for 12"" graders who graduated is students who were enrolled in 12" 
grade during the 2014-15 school year. 


Comparison of Different Positive Outcomes by School Mobility Patterns 


The data on 12" grade graduation rates by school mobility patterns aligns with prior research that staying 
in the same school is associated for students graduating. Students who entered a CGP school during 12" 
grade from a non-CGP school graduated at a higher rate than those who transferred between CGP schools 
or out of a CGP school. 
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Figure 50 
Positive Outcomes Across School Mobility Patterns 
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Note. Students could have a positive outcome of returned to school and stayed in school OR graduated. 
The denominator for returned to school and stayed in school is the full students served 2014-15 annual 

sample. Whereas the denominator for 12" graders who graduated is students who were enrolled in 12" 
grade during the 2014-15 school year. 
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PART THREE: COHORT OUTCOMES 


Research Question 6: What are the patterns among targeted interventions 
delivered in ninth grade, school mobility throughout high school, and student 
outcomes four years after initially entering ninth grade? 


This section of the report expands on section two by describing 
outcomes for students within four-years of ninth grade. The 
outcomes are connected to the targeted interventions that the 
students received while in their first ninth grade year and 
attending a CGP school. 


Throughout this report, differences 
stated in the narrative (e.g., “higher 
rate”; “overrepresented”) were 
statistically significant at <.05 level. 
Whereas language implying rank 


order (e.g., “highest percentage”) is 


Students may have been exposed to or served behind the scenes 
by a variety of methods and tactics and perhaps offered targeted 
interventions in subsequent school years through CGP programs. 
Information on exposure or dosage are not available beyond the 
students initial ninth grade school year. 


descriptive and does not apply a 
statistically significant difference. 


The specific test-statistics for each 
comparison are available upon 
request. 


Outcomes by Student Characteristics 


The outcomes for the Class of 2017-18 cohort were overwhelmingly positive with 72% of students having 
a final record with a positive outcome (Figure 51). Gender differences were not statistically significant 
(Figure 52). African American and Caucasian students had higher rates of neutral outcomes compared to 
the general CGP served student population, whereas Hispanic students had the highest rates of negative 
outcomes (Figure 53). Highly mobile students and Title | students had higher rates of neutral and negative 
outcomes than the CGP student population as a whole (Figure 52). 


Figure 51 
Cohort - General Outcomes at Four Years After Entering Ninth 
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Figure 52 : 
Cohort - Outcomes Across Gender 
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Cohort - Outcomes Across Race/Ethnicity 
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Figure 54 


Cohort Outcomes Across Unique Student Populations 
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Outcomes and School Mobility 


Remaining in the same CGP school had the highest rate of positive outcomes, while moving multiple times 
had the lowest. The first two school changes were associated with an eight percentage point drop in four- 
year graduation rates (Figure 56). 


Figure 55 


Cohort Outcomes Across School Movement Patterns 
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Figure 56 
Cohort - Outcomes Across School Changes 
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Outcomes and Services Received 


As noted in the previous section, applying descriptive findings to programmatic decisions must be done 
with extreme caution. These analyses do not take into account the unique circumstances of the students 
receiving those services. Higher rates of positive outcomes are not necessarily a function of intervention 
effectiveness. 


All targeted interventions, except dropout recovery, were associated with more than 70% of the students 
served having a positive outcome four years after initially entering ninth grade (Figure 57). Students who 
received dropout recovery interventions had the highest rate of negative final outcomes, which may be 
related to student selection (e.g. students who had dropped out). 


The outcomes associated with most methods and tactics ranged from 67% to 72% positive outcomes. 


Family Partnering and Enhanced Counseling and Mentoring; however, were associated with significantly 
lower rates of positive outcomes (50% and 58%, respectively; Figure 58). 
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Figure 57 
Cohort - Outcomes Across Interventions 
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Figure 58 
Cohort - Outcomes Across Methods and Tactics 
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Comparison of Types of Positive Outcomes re eae ae ae 


Positive outcomes for the cohort analyses were assessed students who were served by CGP and 
four years after students initially entered ninth grade in two had a positive outcome. 

ways: 
Positive outcomes, as defined by CDE, 


e How students exited the Colorado public school ae f 
refer to four years after initially entering 


system four years after initially entering ninth grade 


ninth grade: 
e How many students returned to school after a e Remaining in School (Still 
prolonged absence during the four years after Enrolled) 


inlelally Cre nner eta Obtaining a high school credential 
The most common positive outcome four years after initially (e.g., GED) 
entering ninth grade was graduation (56% of students). Graduating from high school 
Another 168 (16%) students were still enrolled in school but 
had not yet earned a high school credential. There were not Or at any point between the 2014-15 and 
enough students who earned a high school credential, such the 2017-18 school years: 
as a GED, to report that information in this report. 


e Returning to school after a 
prolonged absence 


Of note, students who returned to school after an extended 
absence were also counted in final outcomes (e.g. a student 
can be counted in the returned to school category and also graduation). A total of 86 students returned 
after an extended absence between the 2014-15 and the 2017-18 school years. 


Gender differences were not statistically significant for any of the specific positive outcomes, meaning 
that males and females had similar rates of graduating, remaining in school, and returning to school. 


There were however, notable differences in four-year graduation rates among race and ethnicity groups. 
Among those students who were served by CGP and had any type of positive outcome, African American 
and Asian American students had the highest graduation rates, while Hispanic, Native American, and two 
or more race/ethnicities had the lowest rates graduation rates. Race and ethnicity differences were not 
statistically significant for staying in school or returning to school. 


Unique student groups describe their experiences and their eligibility for services. Some students may be 
part of multiple unique student groups. Among unique populations of CGP served students, highly mobile, 
special education, and Title | students had the lowest graduation rates, but had higher rates of staying in 
school. Title | students also returned to school at higher rates than other unique student groups. 


Mobility is associated with the type of positive outcome that students experience. Students who remained 
in the same CGP school had a 72% graduation rate, while students who changed schools had a steadily 
declining graduation rate. Students who moved 3 or moves times had a 28% graduation rate. Students 
who had 3 or more moves had the highest rates of returning to school and remaining in school. 
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Figure 59 
Cohort Positive Outcome Percentages 
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Note. 72% of students had a positive outcome. The percentages reported here sum to greater than 72% 
because students who returned to school typically also have the positive outcome of “still enrolled” or 
“graduated” four years initially entering ninth grade. 


Figure 60 


Cohort - Positive Outcomes Across Race/Ethnicity 
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Figure 61 
Cohort - Positive Outcomes Across Unique Student Populations 
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Note. Students can be members of multiple unique populations and the number of students in each group 
varies substantially. Thus, the most practical interpretations of this information can be made by comparing 
unique populations to the grey bar which indicates the population of all CGP students served. 


Figure 62 


Cohort - Positive Outcomes Across Schoo! Movement Patterns 
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Figure 63 
Cohort - Positive Outcomes Across School Changes 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 

Returned to School Stayed In School Graduated 

m No Change 2% 7% 72% 
w 1 Change 7% 18% 53% 
m 2 Changes 9% 20% 43% 
m3 or More Changes 24% 30% 28% 


Figures 64 and 65 depict the type of positive outcome that students experienced four years after receiving 
a specific intervention or their school implemented a given method or tactic. These analyses are not 
causal, and information should be applied to action with caution. The students who received the 
interventions were likely not similar. 


The most appropriate use of these descriptive analyses is considering the rates of graduation, remaining in 
school, or re-engaging school with what practitioners would expect this approach to yield for the students 
whom the intervention is intended to serve. For example, the finding that dropout recovery is associated 
with the highest rate of returning to school makes sense as that intervention is designed to re-engage 
students who have discontinued their education. The graduation coaching data suggests that less than 
half of the students who were identified in ninth grade as needing graduation coaching, did not earna 
high school diploma within four years and a quarter of those students at a minimum needed more time to 
reach that educational milestone. This raises the question of “is graduation coaching enough for students 
who are flagged as needing additional support in ninth grade?” (Figure 64). 


The crosswalk between methods and tactics implemented at a school CGP students attended in ninth 
grade and those students’ outcomes for years later should be used in a similar vein as the intervention 
data (Figure 65). For example, ninth grade students who attended high schools that emphasized family 
partnering or school climate had higher rates of returning to school after a prolonged absence and 
remaining in school after their anticipated graduation rate. This may suggest that family partnering 
strategies’ attention to school climate may help keep students enrolled in school. 
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Figure 64 


Cohort - Positive Outcomes by Intervention 
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The attendance interventions had the highest graduation rates for students who remained in the same 
CGP school or who transferred between CGP schools. Check & Connect had the highest graduation rates 
for students who remained in the same CGP school while credit recovery interventions had the highest 
graduation rates for students who moved the most, including into or out of a CGP school. Dropout 
recovery had the highest graduation rates for students moving into a CGP school. Graduation coaching 
was not statistically significant when compared to school movement types. 


Figure 66 
Cohort - Graduation Rates for 12th Grade Students by School 
Movement and Intervention 
70% 
60% 
50% 
40% 
30% 
20% 
10% 
0% Zz = | = jt. 
Attendance Check and Credit Dropout Graduation 
Intervention Connect Recovery Recovery Coaching 
m One CPG School 52% 8% 53% 1% A% 
m Between CPG Schools 59% 0% 32% 0% 17% 
m Into a CPG School 42% 2% 63% 9% 2% 
m Out of a CPG School 45% 2% 66% 0% 3% 
= More Than 2 Schools Attended 36% 3% 65% 2% 5% 
Conclusion 


Reflecting on patterns in dosage, disproportionality of service delivery, and student outcomes from prior 
grant funded programs can inform future investments. The goal of this study was to explore — using the 
available student level data — patterns in the CGP program data to make recommendations aimed at 
strengthening the SRGP. Limitations in the available data and the descriptive design mean that results 
cannot be said to be caused by the services. Instead these findings are best used to generate ideas and 
guide conversations about how to move forward with grant-making. There is a clear need in Colorado to 
accelerate progress for Title | and highly mobile students. A next step might be to rigorously test if specific 
grant-making strategies or targeted interventions do in fact produce better outcomes for Colorado 
students. 
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